In order to verify that software is functioning properly, the software must be adequately tested. Software testing is typically done after a threshold of change is reached to warrant the testing. Software testing may be performed during final integration of code or at intermediate stages in the development process, particularly in the case of more complex software packages.
Typically, testing is a manual process during which a set of test commands are entered manually by a test engineer. In order to test a larger, more complex software package, a batch file including a set of test commands is often executed. During a test session, a log file is often generated which includes the set of test commands and associated outputs. Typically, the log file is searched manually in order to identify software “bugs” (i.e., errors) and their possible causes. Errors, such as failed execution of a software package, are not the only software regressions that a test is designed to uncover. Software regressions also include slower execution times of the software package, modules, or portions of code compared with previous tests and/or developer expectations.
A log file may contain large amount of data and therefore searching through the log file for indications of software regressions may be a tedious process. Thus, the debugging of source code can be a time-consuming and cumbersome task.
Various debugging products are available for debugging source code written in various languages (e.g., C, C++). Using these debugging products, the values of various variables may be traced and lines of code may be stepped through to assist in debugging this source code. Such debugging tools commonly accept a program, with or without “source code”, as input and execute it at the user's desired pace, displaying the source code lines that were executed and outputs resulting from these executed lines (e.g., debug information and program output).
Unfortunately, existing debugging tools are only appropriate for testing small sections of source code when many software packages include thousands of lines of code. Some examples of such packages are the Solaris system, available from Sun Microsystems, Inc. The Solaris system and comparable systems include many software packages that have complex interactions with each other in a non-deterministic fashion. Moreover, these types of packages are often built by hundreds, if not thousands, of engineers. As a result, the software changes may come from many different sources. In addition, such software modifications may affect a variety of system components other than the portion of code that has been modified. Thus, although it is possible to test each software change prior to its integration into a larger software system, it is often difficult to predict the second and higher order effects of the sum of the changes made to multiple portions of the software package. Accordingly, it is often difficult to identify a problematic portion of code upon a regression of a complex software package.
Platform variations may also complicate the software testing process. Often, companies develop software to run on multiple platforms. Since failures may occur due to hardware as well as software problems, it is typically desirable to run the software on different platforms to ascertain the source of the problem. However, execution of software may result in a regression on some platforms and exhibit no problems when run on a different platform, even where the cause of the problem is software-related. As a result, engineers are often overwhelmed by the numerous factors that must be considered when attempting to identify the source of a software failure.
In order to identify the cause of a particular software regression, brute force is typically used. One example of a brute force method is referred to as a binary search. To illustrate a binary search, the following description is provided.
The completion of a threshold of change of a software system is often referred to as a “build.” FIG. 1 is a diagram representing two consecutive builds between which multiple code revisions of a program are performed. Typically, testing begins with a code base that is considered “clean” (i.e., relatively free of any known software bugs), illustrated in FIG. 1 as Build A 102. At a later date, a code base producing a regression is shown as Build B 204. Between two builds, there are often many code changes, illustrated as Delta 1 106, Delta 2 108 . . . Delta N 110. These changes may accumulate significantly in a short amount of time. It is not practical to fully test after each delta because there may be a hundred deltas that have occurred within a single day when a full test of the software package may require a week's time.
A binary search begins with testing the software system at point Delta N/2 (not shown). If the same software regression is identified with respect to Delta N/2, then the software system is tested at point Delta N/4 (not shown), and so forth until the source of the regression is identified.
Numerous man-hours are generally required in order to successfully test a particular software package, not to mention running an actual test corresponding to a single delta. As a result, each day that passes is extremely costly.
In view of the above, a need exists for a tool that aids in locating the cause of software regressions identified in a software system.